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. - ■ ' Ihla paper repwts the results of a ' stu% that 
.inteatigated factors constraiMng or encouraging th% use of ^ 
avaluation data by school administtators* It begins with a lengthy 
analogy abotJit the use ^of a restaurant 'guide that is intended to 
illuminate how constraining or encouraging , factors influence the use 
of ^evaluation data* The author defines a constraint as something a 
typical /adttlftistrator iiottld consider a' limitation oh under standi a g or 
alternative courses of actioni The .paper then explains and 
illustrates tiio major conatraints to the utilization of eviluatiQn 
results. The first is ^proximity # " tibich the author definas as 
similarity in tim,e or structure, Hithin ^he author's frametfbrk^ 
^•structural proximity" of evaluation ^ data is ^ the similarity of the 
data in atrudtare to the needs of the educators or the material being 
evaluated* "Temporal ptoKimity" means tha timelinass or ejitrency of 
the data # The author concludes that both^kinds of proximity have^a ^ 
positive influence on data collection. The second constraint - 
identified is compating demand^s on admitoistpators' time, AGCdrding, to 
the authort administrators are unable to pay attention to s valuation 
aata 4ihen' .there are too many bthet demands on th6ir time, 
lAot^or/JM) - ^ 
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Introduction 




A nuitiber ^F iiitg fc^lng' uti'l-T^atMon-related pKttern-s / , 
emerged fNni'^his stiidy's data. /Perhaps the cleafest of 
tKese was the "dlffefentUl Impact of >aifttrcu3ar types of - 
tvaluatli^n data. For exampill, respondents, frequently noted 
that criterion-referenced testa had jreater Instruction^T 
relevance inipact ' than norni-r&ferenced tests and consequeritTy 
their Results wire used -^0 a greater extept. Similarly* . 
the external , unfamiliar nature of a team Veview process < 
vras repeatedly" M amed f or Wts. sfjort-l 1 ved effect --.two' 
wears' after it had occurred the recommendations were already 
tiejng ignored- ' - , • ■ 

. . Closer investigation of these seemingly contradictory,- 

'* - ' ^ ^ ■ ' ■ 

responses to evaluation daita revealed a more cd|pl ex pattern. 

Why is it that irt some, instances a constel 1 ation of 4val;uatio 

factors prornoted attention tb evaluation while in pth-ers no 

impact occurred? It seemed thaf a set 'of fact^ors was preset 

in our data that had either inhibited the attention 'paid to:. 

evaluation or enhanced its use by decision makers. This - 

paper' investigates these p'henopena. Based on the reports of 

principals, coo.rd1 nators , and resource teachers'^ yte found 

identifiable 'fea.tures in Ithe school environment, the nature 
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Of^ ei/iludition tcti vi ti es and* i r\ the i nteract fori between 

' -^^ .■ ■ , . ■ . , ■. ■ ■ " ' ■ ' ' • ' ■■. . • 

fe^Tefnents iri the evaluation process that- constrained or 

' ' incouragM use of evaluation informat^n in- si te-1 evel 
decision making. , " • ^' , / ; ' ^ 

■ • s' " ■ * \ 4 ' ■ ■ ' ■ ■ 

• — TJi-ree broad consteil atlone -of- factors emergfed from 

V ■ '■ • ' ' ■ . ' : ■ ^ ' ■ . ■ ■ ' ■ 

the data. The first consi'sted of ai srqan set ■ of features. 

that |an be c1 ustere^Aunder the heading of ^^'proxlml ty" 

the ^'cloieness'rof the' ^valuatl^n data, to the Inforniatign' 

: i^r^id^ in sectin^ f eature v/as a 

concern about cotnpetingt demands on the time available to 

'• . ^ ' '.s- ■ , ' . ' ' ■ ' • . ■ 

school personnel for- us,ing evaliiatlon data. The third w 

: " * ' ^ ^ d ^ \ 

a complex interaction of a number of p^sHhosoclal variable 

that manifeste'd- its'il f •strongly in the affecti.ve ''realm -- 

pedpW's attitudes' ati^d feillngs toward evaluation. This 

piper reports on *the first two . of these features . ^ture 

reports^Hin describe our flntflngs concerning the/psycho- 

srfelal variables'. 



r 
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This p^er wf;ll proceed in three^stages./ First, we 
will pres^entran analogy to be us#d thrpughout the ^analysis 
to help mumlnate the influence of the conitr'aintng' an^j 
encouraging factori tliat emerged from the data. S'tecond, 
'wef win define the terms "constraint" and "encourager" 
as. they have ..begun to emerge in our study. Finallys our 
analysis of proxim1t)|^ concerns and^competing demands on 
^^^fime w1 ll^be^presented al ong th ' some concluding rernarks. 




i. 




1 t 



AnrEval uatlon Analo 



erJc 



Let us cpnslder^the dec1s1tfn**rnafc1ng' procits ,re 



to 1 Very stnit)la and^personal eyent da^fding Vrhat to do, 
abbut dinner. It Is 18:00 P.M., Ruth Wtll is , ^prJhclpar df . 
the El Martas School joj^t a;rrlved homa and Is fa'cing tha 
diltmmi of ^how to meet her hunger needs* There are many 
actions Rifth^rrtight take. She can go to the kitchen and 
cook dinn.fer vfrom what*s available in the ref rigeratorp 
She mlght^go t^ the store and buy a p^rtlcuTar item that 



jappeal*s *to her and prepare 1^ at^^home, '^She might go out 
tc a ras^taurant and, pay to have someopft ^el se da the prepara- 

- ^ ~- — ^ ' ■ . ^'^^ V ' 

tlons and cVsan. up. There are probably other bpfeti ohs^^as 
v/el l 5 hut^&t us say thgt these are the only thl^ee ^al tferna- 
.tives that seem reasonabTe to Ruth, 

lEarliers the^ di strict pr^incipal * s meetihg .1 ajsted until 
6:30 P,M. rte^ult, Ruth is extremely tired?\she : 

doesn,' t feel I j fee .cookl ngV sl^en going out and buying her 



favorite f i xi ngstr^ems Tike too much v/ork Instead , she 
d-ecidd^s to go\€tf:f^d.^re • • 
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i' ■ , .. , ■ ~:m&. 

: • Ruth recently, recelvl^i^^^ltj^^nt guide as a present.' 
It is enti t1 ed'i Best fb »it iuirants ef Jhe Ci ty f roin $3. 5,0- to 
$50\ per PersQ'n » and is § mthBt thaf*ugh eviiluation oj* pos-J 
slble places to d^e. The questions, of concenn t& us are: 
Ho^ does Ry^th use this ? «rmat Urn? How do the otKer princi- 
pals ^ho also arrived honi«* a* S 00 P.M.^ not v/anting to cooki 
use the information in their copies of the^ popular restaurant 
gu1de-^1n deciding where to eat? What characterlstitCs of the 
inforniatlon are Important in determining fts usefulness to 
Ruth *and her colleagues? Are there certafn feature? of the 
situation that make all the principals behave. in asfmilar 
manner? What contextual factors are the most infltien'tlal In 
promoting qr inHlbiting the use of the guide? M 

Consider what might happen First ^ Ruth checks, her purse 
and di^scDvers tftat she has only $10. This immediately elimi- 
nates a numbar of choices the $50 per person clta^tions in 
the book are npt going to do her to6 much good. As an energy* 
^conscious parson, she doesn't want to drive more than five or 
tei^n miles^ BesideSs It's now close to 8:30 P.M. » so she'll 
have to find a place that serves late. 

As you cart see before she ' s even p1 eked up^ the restaurant 
guide (looked at the evaluation)- a number of factors 1 imi t her 
possible actions and the use she can make of the availably 
information. Resources are an obvious^ concern . v/hl ch effect ^ 
her actions and her use of the information * Her attitude 
toward driving long ^distances limits her choices^ as does the 
time constr^nt Imposed by her particular situation, .At 6:00 - 
P;M* the restaurant guide might be much more useful than it is[ 



at 8:30 P.M. Information, about "The :^ar1y Supper Club" .is V 

• . . '■ ' I » 1 ' , * ' ' • .' ■ ■ . 

totatly useless to, RUitH'-^al; this point .irt time. . ' ., 
Both Umite'd Resources a.ncf, Tack*of timeTy access^"^ 
dita are factors that have beeji Identified in the research 
as 1nfluences«OT evaluation util Izltion . Al kin ,i Dai ITak andf 
White (1.979) noted the Impact of fiscal constra fnts and * " . 
ida'ntlflad this as a prop^-rty >/i^hin thair framewoirk. Time-- 
11ht>s has been Identitied by many writers as^ being an*. 
Important varlablt ,^m'u'til izatlon, <Alkih, 1975i Cohen, l9'77| 
Mitchen, 1373).^ ; , . / . - 

There «re <ther 'constifa^nln^ factors, to -be found in this 

■ ' . ■ '^v " ^ ,^ ■ . ' " ^ ^ / / ^ ^ 

^e^mple; two win suffice, for tH ustration:^ Ruth' s attltiides 

y\ ^ ^ ' " . / - ^ /; ^ .'^'^ ' ^ ■ ^ ' . ' ^ 

toward travel is constraining; she. Eoesn ' t 1 ilce to drive far* 

anid this^ Units how>she might use the recommandati ons to be 

;^QurTld in the book Simi 1 a Hy , ^1 f^ she disHked I ta^ tan ,f ood ^ 

a whole section of the book might bB of ho interest, 

. . Thl first/two features we tdent4fi.ed,i^- the level of 

available ri^oUrses and the restr 1«ct1ons Imposed by the^ clock 

--^probably affect al 1 the pri nci pal s. in the ^district in a 

> -■ . 2 ^ - ■ " / ^ ' 

sijniTar manner. On the other hand, Ruth's f eel Ings about - 

dri ving and about^ Ita3.1an food are clearly her own* Other 



"Later 1n thir paper^ ^ will ^ distinguish between two differ^ 
ent aspects of timeliness: 1)' tiie passage of time- del imi ti ng 
courses of .actton ^ ;and4 ^2) eval ulation in-^rraation riot 

^available at the time-^that a d'sf in^i te deci sion rrjust be made. 

^(thTS w^uld be the case if Ruth's, friend gave her the ^uide 
nj^ week instead of 1 ast \veek 1 ) . ' . 

^Otte^'must of course allo-w Jor individual ^differGnces\i^i^ is / 
,i5ps"l1^e that one of the; principal iS independontly wealthy 
and more .^f a' ''nigKt person*'^than most of his or ^er 
'col 1 elguars . v . ' ^.^ ' . /' 
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prlncipaTs are 4bss J i kely to share these opinions. As a 
. consequince they ^mig'ht have more options op^tn 'to them and . 
fnight/find the information in' the book more useful ,' 

This contrjist highlnghts lan importaftt aspect of ouri ' 
analysis.f ^ " ^ , . 

1) Personal varlaBlls enter* into almost every decis^ionr- 

* . 2) Jhpre may be jip such thlngs/^as universal encouragers 
or constraints. *As a coniequence>-Our^ labelling witht 
these categories wilT be based on widely prevalent reports of 
factors that inhibited or f aci 1 rtated utilization^ not 1/ 
unanimous bptnl*ons, V * 

Let US, now exam1ni[ some of ' the gu Jde " s featuresyv/hich. 

tended to encourage its use in, Ruth-s decision.^ FirUt, the 

^ — . — _ _ ^ . N 

guide has three ^different cross referenced. Indices * Rtstaurants 
^are-listed by |eographic areas by. type of food and by'*price. 
ThuS j when Ruth opens it ^ restaurants are readi ly identified | 
whf/ch f i t' her requirements an inexpensive, local place to 
^eat.^ Secondly, in addition to its lengthy descriptions of 
e'^ach es'tabii shment the guide al so |ias a very simple abbreviate) 
rating system. A 'quick glance at^the top of eadh page pro-^ 
vides an instant summary of '-quality of food" and "quality of 
service,*^ Also, the book is quite ex^tensivej providing in- 
formation on a "large number of choices ^ Flnally^^ v/e musn ' t 
forget another fact v/hich may encourage Ruth to pick up the 
guide and use It in making her choice. , .hunger. ^ Her subcon- 
scious needs assessment tells her that food is her number one 

* priority for the next hour or tv/o. 



lest th'e analo,ay'be unbalanded, we shou 1 d 4l so. point 

' ■ . ■ ■ • ■ ^ ■ ■ ■ ' ' . '. . . . . ; . 

out't'hat the fea.ture-s j.ust mentioned do not all act as v. 
-«^. , ^ V . , . , % , , ■ . . , 

• Bncouragers to every principal . Hunger, -for exaniRle, causes 

some of Rqth's colleagdes to ignore.the guide completely; • 

revj'ewing all that data' 1s, simply too tlme'^consumi/lg.^ Others 

f.ind the book's bulk overwhelifiln-g; It offers too many choices.' 

}' Long before systematicaVly reviewing a^- their a'l teVnati ves j - 
they throw^lt^ down ,on the coffee table aad head for a fam'il- 

' iar,"d1ve." Thus, for some people, hunger acted as a con- 
straint ^/hile for others 1t was an encourager. to the book's 
use. Likewise, the sHear bulk of the book served to constrain 
only some of the prlnftipals. , ■ - 

Research h-as addresled some of, these factors. The Impapt 
of reporting format and complextty h^ve been examined by 

' Alkiri et al. (1974) and Glaser and Taylar (1975). On the 
other hand, the perdeived' need, for. an immediate decision has 
not been subject to a gi-eat deal of resaarch scn.ut1ny. 

Let us continue with our story — what about Ruth? She's 
reviewed the guide wi some care #nd made her choice. At 
9:0d^P.M, shs^finds herself, along with many of her Gonefgues, 
sitting at an all nlgh^t dellGatessen waiting for the surly 
>fai tress to take her crder. In thls-casep most of the princi- 
pals came to the same decision. Similar constraints and 
^^aluation information |ead them to take the same course of 
action , » 

But v/alt, a fev; of Ruth's f ellov/ princi pal s are missing, 
Whatf happened? One principal is eating a $50 steak and lobster 
combination --he found the resaurces or maybe he used a credit 



:nd 
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•card. Another principal ;||. at his local haniBurger stand. ; 



He recognized a limitation of the evaluafion ft didn't, 
iticlude any restaurants less expenslvi than $3.50 per^ ^r^son. 
Oni prtnc i pal was remi nded by sbrnething In the giJide of a' . 
new restaurant that just oprened around the corner from his { 
house* and. he went there for dinner* Another noticed that 
the book was a year old^ so he looked -for suppl^emental refer-' 
ences in the yellow pages. He discovered a new Vocal restaur- 
ant. Finally^ one principal said ^ "Its too damn late to, go 
•outt and I'm too tired," He went to bed without dinner,; - 
(Illustrating that even needs assessments bave 1 1m"i tatldn^ , ) . 

This analogy highlights the multiple decisions that 
might be arrived at based on the same Gvaluative dfita, ^Also 



illustrated are sorne of the factors which influence individual 
J ' • ■ ■ ^ . - " 

patterns of thought and action, *As we proceed through the 

analysis of ths data from this stuSy we will refer to the 

c1 rcumatances prasented' in this example. The analogy will 

help muniinate some of the distinctions that emerged in our 

investigations, ^ ^ - 



Constraints and Encouragers Defined 

Ruth Willis- dilemma provided a g^od opportunity to 
introduce , the concept of constraints and encourageps to the 
utilization of evaluation. A number of interesting features 
in her situation affected the likelihood tha*t th^, avai 1 abl e 
information in the restaurant guide would impinge on her 
dinner decision (e-g% aval 1 ablo resources ^ tinsi clarity of 
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da.ta). An<J to some extent^, different pattern's^ 

between the pf Inclp-als .' : , ^ " • h ' : 

Our study produced stmi lar rindihgs. While all the *^*> 
^^hools in the study ha<l access tO/ very similar evaluation data^ 
difftrent pattern^ of use emerged. Yet within these dlfferenGes 
certain contextual! featftjres cm^-up repeatedl|^ in our convers- 
ations as irnportaht in determl ning declslorv rAakers' reaction to 
and use of Information avanable to them, , The notloh of con- 
stralnts affd enGburageys was borne of these similarities* \ 
We will refer to something 'as a const raint if a typical ^ ^ 
nlstrator wouTd f 1 nd thi s feature a limitation on his or 



her understanding of a situation or his or her alternative 

courses of action. i , 

^ .■ . ■ ■ * ^ ' ' ^ ^ ' ■ ' 

Not all persons would necessarily act in the same manner 
given the sanie circumstances. What one person perceives as an 
InsurmountabTe obstacle to some course of action might be per- 
ceived by anothar person as merely an inconvenient nuisance. — 
» ■ , . ' ^ =■ ■ ■/ ■ 

This points up the fcfct tfiat the notions of constraint and 

encourager are rel ati ve , n^Dt absolute. Thus, our use of the 

ci — ■ ^ - ^ 

rterms --constraint" -and. "encourager" in this report is normative 



The quest1on;may be raised why v/e bother to specify both ternis 
(constraints encounagel^) v/hen they are apparently opposites^ 
and a *si ngl e def i ni ti on might suffice. There are ,two re&sons. 
The first is data based --they were viev/ed as distinct enti-- 
ties by our interviewees. Administrators themselves saw certain 
' features as TimHing and others as enhancing. It seemed worths 
wh41e to maintain this distinction in our analysis. The other 
'advantage for creating both^labels is that certain si tuations 
are easier to descrtbe from one point of view rather than the 
other. While this is purely a syntactic conveni encc , wo- de- 
cided to retain ft because it v/qs so easily accompli s lied . 



We will onT)^ put iorth as constraints or ehcouragers those 
features which represented a substantial comnvonal Ity across . 
interviewees. ' / ' *\ * ' 

* A final note seems in-order before descri b4 ng oyr study 

f • . - • . ' " . ^ • ■ ^ ^ ./ ■ . 

' • • - ' * " .i * ' 

results. By focusing on features that v/e,re comrnonly 'seen to ^ 

be constraining or encouraging, we do' n^t mean to uhderistimate 

the creativity or individuals InHtlatlve of schpoj^ admihi strators 

The respondents in our study were a heterogerieous groups and • 

there is probably an exception for every general ization we 

will offer. There ^were administrators' In our sample for whoni 

even the most frustrating circumstances were not allowed to act 

as constraints. Such especiaTly creative Individuals are 

probabTy worthy of a'dditional study themselves^ Whether you 

characterize thsm as creativtfs stubbarns sel f ^centered j dynamic 

or as troublemakers 5 they were often unconstrained by factors 

that inhibited "iost of their colTeaguej. They are the outliers 

in our study^ and 1 fke outliers in any data analysis, they 

should be Investigated more carefuny in the future. ; : 

... ' * i - ^ ' 

Constraints and Encouragers in Practice 

At this point we begin pur discussi of the practical 
results of the study. The features we will identify as con^ 
straints and encouragers are constructs drawn from the descrip"^ 
tions given by the respondents i n the study. They appeared 
as recurring patterns of events and understandings reported^ 
fay many different re^pohdents ^at 'rnany differenb schools, ilt 
is this general character that gives them practical signlfi- 
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canct. We have seTect=id a wide variety of quotes from our 

respondents to inustra'te their points' of V^ew^ 

We begin with a dlscusslori of the feature we have called 

proximity . ^Two types of proximity will be distinguished 

structural and temporal . Then v/e will di scuss competl ng 

demands on time and the manner in which such demands act as 

a constraint to the utilization of evaluation. ■ 

Proxlnii ty ' 

* We/use th§ word proximity as a generalization of the 

notion of distance how close or far* away one thing is froni 

another, ,Jut^e use the term to mean more than Just a spacial 

comparison. By proximity we mean the degre6 t hich tv/o 

things are similar or dissimilar along any number of different 

dimensions.' Of particular interest fn this study are the 

dimens.iohs of time and structure v ( 1 ^ e , form ^ sty 1 e ^ content* 

'etc.). "^Consider the follov/ing comments offered by three 

responuants in qur study:/ - 

"Sonetinies the district sends us eval uation forms v/h1ch 
dOn ' t real! j^Mieet w hat our school is doing; we try to 
devise our omi based on what they've given us," (llSPl) 

"Well 5 for the teachers who really are involved in using 
the test scores from state and mandated tests its help^ 
f ul . . , and v/e do have a couple of teachers .>/ho use that. 
But mostly our teachers use the tests from XXXX (the 
management system) t^e math program and from the reading, 
program,' They mainly use those to see v/here their children 
are and to replace and regroup." (13SP1) ■ 

"It seems to me the district^ needs to get i nf orma tion ^to 
the schools more quickly on Issues that affGct every 
single classroom teacher, v;hich moans those issues af^ 

'^^ fecting every single chTld within those classrooms.". 

V ^ (ZOSPl) , 

In one, form or tinotfier, thesr three respondents are all 
talking about the Stinie tiring, trii) proxiinity of the i n rotMi?a 1 1 on 
to some dc'CisioiK Tfie greater t!ie UlisLanco* botwoen the - 
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inforination and an action the less likely it is that the 
Informaii'on v/Hl ,be considered in the decision on hoy. to 
act,. The more effort that is required to translate., inform-- ; 
ati on. i n tb ji^ useabl e form^ retain it until an appropriate 
time or slri^^^t of^^motional overtones t^e less likely it ^ 
is that theie trfinsf ormatioas wi 1 1 be made. In our research^ 
data related tfo ^proK^fn^^tyJ^eimed to ^ b easily categorized - - 
into two types: Structural, proximity and temppral proximity* 

Structural Proximity / ^ - . -. 

Structural proximity denotes t'he degree to v/hlch a new 

element matches the format 5 content , or style of the existing 

4 ' 

elements of system. 

Respondents in our study commented frequently on the - 
form and content of the evaluation information available to 
themV They reported that the configuration of the information 

whether it wis directly usable for teachers- instructional 
decisions affectjed Its utilization. 

This was manifest among other v^ays by comments on stands 
ardized testing. and tKe districtVs new criterion-referred test. 
A large number of 1 ntervi ewees attested that standardi zed tests 
were less useful than local w1 th in-school tests that were based 
on the school/ s Instructional program. A second common observ- 
ation was that 5 among the requl red achievement tests^ the 
district's criterion referenced test (DCRT) had the potential 

'^A further subdivision Is possiblG,; One qoulcl differentiate be 
^tween the form of information - - r its physical arrangumont 
and Its^content. Wh i 1 u /ev i done e for sudi dffrerchcos can be 
found in our data, /it Is a fairly technical dist^inctlon wfiich 
was no t qfMieral lyj.iada. Fon tli^^ time belnii we will consider 
only the f)enVral chj^egory of ^ t rue turc and not divide things 
furtlier. 



be'.niuch - more bseful than the dTBS test. ^ . ' 

- "We use the XXXX reading program, and (evaluate based 

— on) the movement in terms, of the number of , steps 
children achieved during the year... The XXXX is niuch - 
easier for use because there * s a da>ly, even a weekly^ ] 
evaluation There are so many yariables in a one. 

shot test like the CTBS ^ so f rom the schooT's point 
\ of view the XXXX management system .(is) much more 

useful to us,," (15P) 

"Why are vve putting up with thi s ( s tandardi zed test) 
year after year, when we know there are better things 
we cocild be doing with our^ time? There are other 
Intruments possibly which we could be using to give us 
the kinds of Information we want. That's why we lean 
more heavily on teacher evaluations and 'those kind of 
/ in-hoOse tests. " (11SP2) 

"The test scores we utilize have been the ofies that are _ 
^ = criterion referenced tests like DGRTJ' (20SP1) 

The important point 1 1 1 ustrated by these comments does * 
-not simply pertain to testing but rather to hov/ easily the 
information could be used for instructional decision making. 
The extent to v^nith new information (particularly from 
evaluation) corresponds to the format and. content of inform-- 
ation already used by the classroom teacher dictates the^ 
degree to which it will be used in decisions. 

Recall our analogy. Under the given circumstances Rpth 
Wi 1 1 i s woul d be much less likely to review the restaurant 
guide before choosing a place to dtne if it lacked cross- i 
referenced indicies and if there were no easy-to-read summary 
ratings for each restaurant. These physical features (the 
form of the guide) made it more useful to her. Under other 
c i rcums tances* s he might be pleased to brouse leisurely through 
lengthy reviev;s, but in her current operating procedure 
(rushed) she responded to a more sparce and efficient organiz- 
ationf 



• ■ ' ' 15 '■■ , 

Similarly, undar those Inim'ed ci reuiiistancos Ruth v/ould ■ 
be more likely to revieV/ a restaurant guide'- that featured . 
only inexpensi v,e^ res'taurants than an encyclopedic reviev/ of. 
.'all dining establishments in t^ ^rea The former .{though ■ ; ^ 
much less qomplete) corresponded more closely to her-ffSpeci f 1 c', •* 
needs. It provided the kinds of information (the content of 
the guide) she v/as ready to receive. ' j ^ ^ » . 

if i ■ ■ = 

An .exaniple from our study would be useful herei A * 
number school S have adopted the XYZ management sy tern to 
coprdlnate their arithmetic program. Students progress Is 

monitorsd aqainst the XYZ ar1 thmeti c conti nuum 1n al 1 - cl assrooms 

• " ■ _ I 

The continuum includes basic ^arithmetic skllis for gra|tfes 1 td^.: 
. 6. Learning tasks are proscri bed according to diagnostic , :. 
^test, and studsnts progress through the skill areas one by 
one. Periodic tasting is used to ver1 fy th'e students mastery ' 
of s^kills and assigri new 1 earni ng jtasks . 

la the fall of the year teachers^ at 'these schoo^ls? receive 
the' a ri thmeti c test scores from the annual Title T evaluation. * 
The CTBS test is used in this eval uati oris #nd the teachers ^ ' 
'receive grade level equivalent scorq^^jp^,^ach student in the 
areas of Computation, Concepts and Applications, 

It should not be surprising to 1 earn 't.hat this information 
is" not very useful. There are a numter of reasons for this. 
One prominent reaso^ is that the CTlS scores have little if^ 
•any direct relation to the XYZ skill ^levels. 
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The information that the teacher receives from the^ eval uation 
based on the CTBS is different, doesn't fit into her regula^r 
pattern of assessmenfT* doesn ' t have a naturaT correspondence . 
to the ongoing program/ etc. It is dissimilar in many respects 
from the ^xisti ng classroom structure and each , of these dissim- 

. ilarities is an obstacle to its use.' At leas,t that is what 
res'ppndents in^^^r sampl e-^'seem to be saying. 

It should be pointed out at this poin.t that we are, not 
critici.zlng the ctBS test on technical ^grounds — val idity * 
and retii^biiity are not the' current issues , of concern* Rather 

,we are s inply noti n-g that this jtest (and other? like it) is 
less likely to be incdrporated into teachers' planning and 

•decision makina if it differs ■ mferkedly from , the^ data the 
teacher is^al ready set^ process* 

Vie can think of several factjors eKplaining why structural 
proximity mnght enhance the utilization qf evaluations. Eval-/ 
uation infornation that has structural proximity is preferable 
because : 

1. It is familiar and therefore more credible. 

2. It requires less effort to translate into a useable form 

3. It matches other data more closely and thus fits more 
readily^into an ongoing aggregate of evaluative data. 

These three factors are affirmed by the comments of our 

res pdn dents : 

"I think that It ( school -1 evel evaluation) is more positive 
because its at the grass roots. Tt * s more bcnef i c1 c\ 1 ; 
its more mcon ing ful ^becauso it takes ploce where the action, 
is../ The initial evaluation in a scliool is teacher-pupn . " 
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"They (the results flrom DCRT) are individual, and th^y 
'are:*the skills the child needs. But they co^e back ,td 
us in a fdrm that^ is not very useable;; In order to get. 
i the tnaterlal i n^^a^ useable form it tal^eS much of 'the 
teachers t Ime / 'and s he * s j us t .try 1 ng- to survive and • 
doesn^f qui^e have that tlmr.^* {14SP1} % ; 

think for t>^ most part that ^the data that ' s ' being 
collected f^^on-s1te programs School Improvement^ 
'T1 tl e. I^ li 5 for the mo s% part ^ useless from one year 
to the next./. The continuum? , the district changes . 
therq. Title I changes their requirements. So from one 
year to the next, the only continuum at^hls school that 
is the same from three years ago wh,en I was Title I 
Coordinator until ^nov/ 1s the one for (ibe readi ng pror 
pram)." /(15SP1) 



One might ask how It is' possible on C broad basis to in-- 
crease proximity of evaluations when each teacher has a. unique 
structure in his or her classroom? How could one ever hope to 
match each of these different structural patterns? Fortunately 
tha simil 1 arl t i in curriculum^ textbooks^ and tests are much 
greater than the differences. Whi 1 e indivi dual . teaching sty les ' 
anV personal 1 tle^ are not stan'dardfzed ^ curricula,' textbooks; 

" ; " ,\ t . ' ^ \ ' ■ ^ 

and management ^systems usually are.^ In most schools ^ each 
teachbr In a grade level uses thje same test. Often the whole 
.school ;,pses the same series. In terms of H nstructional patterns 
book-s, management systems and assessment ces sirni 1 ari ties 
far^ outweigh differences. However^ much greater differences 
occur from school to school within a district such^^; Metro 
(and we would warrant other dist^ricts as well.) 

'It is possible to consider stru^M^al proximity when 
developinq an evaluation design. But^an evaUiator concerned^ 



Ubout utillzation'^nius't make Gertain that t^ie focus is -on 
individual, sch9£l s ^r^ther than some district lov/est common 
denominator. If school level utilizatfon is valued, then ' , 
structural proximi ty concerns^ d1 ctate unique attention to 
each school * s speci fj c educational program. ^ ^ ^ ' j ^ 

Tafiiporal yPrOKimi ty . Concerns aifrout time and timeliness 

r. — ^ ^ ^ - - - - - - 

were mentioned frequently by th^ decision jnakers in our study/ 
Such comments are not -surpri si ng^ since\^emporal ^concerns have 
been^identified as iriTportant aspe-cts of utilization for almost ', 
as long as researchers ;have specul ated about this issue. ^ / 

We alluded to tv/o different kinds of temporaT cbncerns 
when presj|nt1ng the restaurant guide analbgy^ The firit of ' 
these^ timel i ness ^ is used to^ des^cr^^ the Correspondence 
betvJeen the ^reciipt time of Uval uation and the time at Which 
ad^flnistratlve' actions €ire#taker]. In Ruth Willis's case %he ' . 
1 nf ormat i^on in tbj* res taurant guide, was timely bfecau^e it v/as 
a^ilabie'at the time she wanted to use, it. It iWs not still 
%t her fri endj s , house waitings tp be wrapped and de leered. 

The bili^ngual cooi^dinator at one' schoQl" emphasized the i ■ 

importance of tfmellness when discussing annual achi evement 

i • ■ • ■ ' ■ 

tests . * ,. - - > 

' ' ^ . ■ / . ■ ,. '/ ^ ' 

^ ^ "They- aren ' t ^s^ful. I don^^t see teachers using ihem. 
- You |et ^htfti late in the years when you've already 

pi anned *^our^ program . You know the children by then \' * 
so they ^on't give-ydu any new information. "(1 9SP2) ^ 

There is little cloubt froiti our dat.a that prqper;fit between 

del i very * of eval uati on inforinatlon and the action schedule affect 

Hov/evGf, there nmy bo other ovorrldirfg reasons to ignore 
structu>cif— pi^xir.tity . For excUiipTe, .certain standardized tests 
way have general c^nihiiin i ty acceptance that calls for'their ' • 



utilization. What fs ijite.restlngv to note is that some o^f the 
cl eares t ;exampTes of go'pd fit we?:e unplanned, almost coincidental 
Our description of the" fol loiv/ing inGident illustrates the point 
^Aery will ^ ^ , ' * . * * ■ \ 



dne of *the schools observed (SchooT #14) chose ta use 
soniie tff Its special prog^re^m'^f undi ng to create resource 
teacher positions, .Currently there are five such indi- 
vldualSp a reading resource teacher, a math 'resource . 
teacher a mul ti -cul tural resource ^acher , a bilingual 
resource teacher and a language arts resource' teacher . 
Most have their own rooms or Ifirtrsjin whiqh they teach^ 
small groups of students on*^ a pull-out basis throughout 
the schbol year. ^ ' 



As the program at thf school grew In size and complexity 
probl ems devel ojedf-^ tli*6 amount of time certciin children 
spent away from their regular fnst ru^ti onal ^program and 
l(^t continuity as well. \ - * - 

Based on the concern that students were losing continalty 
of instrLi^^i on ind that the1n own workloads v/ere unmanage«' 
able;^hs resource staff wenfc. to th# prin,c1pa^ to suggest 
a^ change/ They^ suggested that resource teachers be ^s-- 
signed to pull-outs with only certain gradie levels,. f 
For exam^^a, th% readi ng resoupce-^eac^er might be as-- 
sicnsd to ihe second anS third grades fo^ the fiPst half . 
OT the year.. Durin| that time no ^ther "^grade level c6utd 
*i send studtntst^ this resource teacher. ^ Each grade ^evel 
wauld h*ave at least one resource teachen asslgheM toMt, 
but none would have more trtali two. 
/ ^ ^ , / 

' Such a pi ah was* agreed to in the spring before everyone 
left for sunlmer vacation^ but no specific assignments 
were made ^ *awa1t1ng the 'start of s^phjpol in th^ fall . ' 

Where does evaluation enter into this decision process? 

WeJJt^actual ly 5 evalyation enters only incidentally. Evaluation 

was not planned as an input to the dec^ision making, but its 

impact V/as nonetheless clear and direct. ^ 

Te^ results were sent back' to %he school In late summer. 
Thf pri*nc1 pal , ^who works year-round, had time^to review 
Hhem, They Indicated that the third and fourth grades ^ 
were scoring particularly low jn the area of matheniat 1 cs . 
The tests did no;t gi^vc too niuch informat i on ^about why 
thi s dcfi c i ency ^exi S tec! or what to do about it, ^ut they 
did pinpoint t hr^^robl em , 
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In the fall the p^rincipal met with the reso^urcc teachers 
to make the grada Vevel assignments . The math fesoarce 
teacher v/as assigned to grades three and four, based on 
the DCRT scores received during the summer: Kindergarden 
N and first grade reading^^cores were low, but t>iere waj a, 
feeling among the group^that these, students were too 
. young to b,e pull ed ou\ of cl asses in that^annerj so the 
reading resource teach\r was assi gned somewhere else, ^ 



This evaluatlqji 1nforma|l on - f1 1 perfectly into the decision 
matrix that had already been e^'stabl i shed regarding resource 
teacher^ al location , There were a limited number of possible 
ways to assign th^^^es4urce teachers according to %the scheme 
that had been adoptWId, W in the fallp all 

options were still open^ and all o^^'ions were equally easy, to 
implement. ^Data directly relevant to the decision was beirfg 
provided' from evaluative sounce^ and it was an iniaprtant 
factor in the decision. 

Thi s exa:Tipl 3 illustrates the fact that serendi pi t ious 

■ i * " . . - - _ . 

connects^ions faetwean evaluatl^on and decision making do exist. 
It would be incorrect to suyfFst,^ however^ that this was a 
conimon experience. ^On the contrary, our research, suggests 
that such occurrerices are Infrequent. More ii^ortantly, from, 
bur conversations with decision makers^ we received t^e clear 
impression that neither evaluation nor . program deci si on^maki n^j 
ere planned to increase the chance that this fruitful inter- 



section of events would occur- 



I ThiwS example aiso illustrates that "a dgcision" in fact 
consists of a number of actions or consecutive "mi n i -deci s i on s . " 
In the case initi^al decisions \f^e based on factors such as 
observations, percoptions j/psycfiological beliefs and personal^ 
ity considerations. Subsequont>e^, the p; ■ ncc of timely cval- 



^Thi5 is quitn coripntible 4'/ith the notion of ^'decision nccretion 
advanced by Weiss (1980). . r 



^21 



lis and his or her old scores on earlier 



. uation data provided the basis fo> a fufther decision increment. 
A- secotid aspect of temporal proxNiTii ty tha t we noted in our 
intervlev/ was the degree to which data are available for use 
within'^their activt time frame. Some furtheT explanation is 
fequired. Most data have a limited lifespan^ and it serves Jittle 
useful purpose to base decisions on them'past their ejjpiratlon date. 
For example, achievement scores only remain timely fon instruc- 
tional decisions for a short peri od ; wi t hi n a month or tv/o the 
cttild has learned ski 

skills are much. less useful to the teacher* 

This* seconi^ type of temporal proximity is illustrated ih 
our analogy by Ruth Willis's using "the most current^dftion of 
the restaurant guide. If one v/ere trying to maximize the djse- 
fulness of a res.tauraflt guides one v/6uld make sure that the most 
current edititn was be^ng consulttd^ not an earlier printing. 

The two aspects of temporal proximity are distinct. It is 
possible for data fro arrive at the proper time fora decision 
to be made (temporal proxjnilly in the first sense), but for the 
data itself to be no longer re/eva'nts, In this cases data about 
students' status'at the time the test v/as administered may be 
useless because they do" not reflect students' current level" of 
performance, > 
♦ The respondents in our study, felt that they 'were burdened 
wtth out-of-date data which was of 11"^le use to them. Often 
they v/era asked to maintairi and pass on data long past its 
usefu 1 1 i f e , 
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"I'm saying that as far as an ovefalT tools to put a 
great deal of faith in^ I don't think it (test data) 
is vforthv^hi 1 e , ^ For the short run . . . v/i thi n a two-v/eek 
>time limits it's a great tool to take a look at.,. for' 
the individual teacher who is working v/ith the class, . , . 
They know who the children are.. .(But) the teachers are 
concerned abotjt, * Why am I passing this data on? The 
teacher next year really isn't that concerned with it 
once ttiey start working with those students."'* ^(ISSPl) 

^ * 
Thus two aspects of temporal proximity .emerged from our 

data,^ Tijnel 1 ness refers to the degree to which data are 

available when decisions are being made. The notion of active 

time frame refers to v/hethetf data are available for use while 

they are still relevant. 

What does this mean for evaluation? ^ First, to increase 

utilization v/e rnust strive to coorinate data v/ith decisions. 

Th\s is no easy task. Major decisions about the organization 

of the school program^ assignment of staffs grouping of students 

in cl assrooms , adopt 1 on of ^urri cul urn- and instruct ion a 1 programs ^ 

etc. occur Infrequently. , They do occur according to a more or ^ 

less predi ctable schedule and some coordination with evaluation^,^^^ 

should be posstble. Minor decisions relating to individual 

students and instruction occur in classrooms every day. Rele-^ 

f ^ 

vants accessable data can be qui ckly uti 1 ized. - 

This reminds us of the second issue. To increase utili^ 
zatlon'we must also strive to make data available quickly, while 
it is still meaningfuK For instructional decisions within ^class-- 
rooms this is particularly 'important, but it shouldn't be ignored 
i n e 1 1 h e r c a s e . ' 

Time can act as a constraint 1n another way, Tho pressures and 
demands of other activities can reduce the a va 1 1 abl e time for 
consideration of data. Howovor, wo do not consider this as an 
element of tcmpora 1 pro;^1mi ty rat her* we will discuss It below under 
the hpriflinn " Cnrnrip h i nn Demands on Time." ^ 
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It is pro-bably best to say that the evaluator who is concerned 
v/1th utilization should strive to increase both types of 
temporal proximity. ' * ' 

To conclude our discussion of the first broad category of 
constraints, proximity, we note that respondents made the identl- 
fications we have descri bed in, this secti on clearly and di'^stinctly* 
Little interpretation nor elaborati on was required on our part. 
There was wide agreement on the importnace of structure and time. = 
The conclusion we draw based on our datm Is that Increased 
proximity would likely have a positive influence on utilization. 
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Competing Demands on Time 

Under this heading, we are referring to the cdnstraintrig 

^presente of competing demands upon deci*sion makers' time. When 

we asked our respondents hov/ much atten ti on they paid to evalu- 

ation, they reported that there were just too many other things 

demanding their attention to focus extensively on the results 

of evaluation. As one ^principal confessed: 

"Well, I'll be real candid with you ^ I get so busy I don't 
pay as imuch attention to evaluation material ai I show 7^* 
I get r|port after report I try to get the general 
gist of c^a.t the evaluation data Is ^ but I do not spend 
a lot of time analyzing it^ and I probably should^^^ 
I think it's probably very good data,. There are Just so 
many demands on me*" (15P) 

There is little doubt that t^e rapid pace anf constant 
^^pressure of the school environmetit constrains admi nistrators 
v/iningness and abflity to devote large amounts of time to 
serious review and analysis of data,; Most adrnin i s t rators in 
our sample repprted being innundated by bureaucratic tasks and - 
political pressures* There were some exceptions individuals 
who purpDsefuny guarded their role as educational leader of the 
school and protected their time, allowing thenrtselves the luxury 
of comtempl ation and forethought- But, by'and large, most of 
the administrators v;e talked to v/ere caught up in the hectic 
business of running the educational facility, keeping up the 
regulations, being sure that the school's program was in com- 
plIancG, supervising applications, attending meetings, learning 
about changes and modi f i cati ons in rules and regulations, main-- 
taining contact with the conimun i ty , supGrvisiTVg'.^discipl ine 5 and 
much mq re * 
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Not only did admim'strators feel th-ese 'pressured , but staff 

did as weTl . Indeed teachers and- a'dmi ni st rators both believe 

that the teacher's job is extremely demandi ngv and that teachers- 

have little free time. Here is a typical description, vn'th a 

suggestion for ImprovemenJt, - ' ' ' : € 

' , , _ J- ' ' , ' ' ■ • . 

"I-m sure you must be aw€re of the fact that a teacher's , 
day is really horrendous in terms of the demands ori that ; : \ 
teacher's time. (Teachers need free time to think) ■ ' 

Industry has learned this I guess we ha.ve learned it , 
toOi but the price tag makes it prohibitive/ I think if 
we^^uld run one pupil -free day a month^j or if we could 
hav^e two pupil-free afternoons a months or if we had an 
% opportuni ty ^ to meet together; and tg interact and to di,- 
alogue and share ideas and concerns wawoul d see improve- 
ment. But the time constraints are such thtft 1t*s liter^ 
ally impossible." {13P) 
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are judgments made and actions taken in— the absence 
of time for con tempi ation? They are made much in the same 
mariner as Ruth^ Wi Tl 1 s mi ght select a restaurant If it was 8:55 p*m, 
and she only h^ii five minutes to choose. She woffldnVt have time 
to look at the restaurant guide and wolild probably^alT back on 
past exparience and familiar patterns of actions. Under the 
pressure to make a quick decision, it is doubtful that she ^ 
challenged any of her previous be! i efs about'which restaurants 
were good. More often than not^ people act conservatively 
in such si tuati ons 5 f all ing back on patterns of behavior that 
have served., them wel 1 in the^past. - . Oskamp , 1 977) 

If the demands of the job act as a constraint to utilization 
of evaluations is there something that can be done about 1t?^ 
Many of our respondents felt there was a solution. ^ Without 
specific prompt ing,- a nu/nber of decision makers concur v/ith the 
principal Just quoted, Thay bcl i eved qu i te firmly that improVGdJ^ 
use of cvflluation data v//is possible. All that was lacking was 
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the opportunity to put. some effort fn the right dire,ct1on. 

Here are two further opinions on this Issue: ^ 

"(If we are going to do sornethirig wfth evaluation data) 
Days have to be set aside. . .if we could have a few days 
on the side where the teachers at least sit down and 
break bread together, I think we'd accomplish a lot more.. 
I don't think there 1 s enough time In the school day to 
have teachers meet and evaluate the school program, 
r think if we had some clear days ahead we (would) Just 
sit and talk, one to one, so it's a group. Group dis- 
cussion to me is the best... I think we need a few days 
without the children available (to) just sit down and 
talk about programs. (25P) 

"Evaluation tells us where we're going and what we need 
to, do. I think It's very impdrtnat. I feel that per- 
sonally I would like to do a lot more of it ... But our 
problem here is (enough time for) iffeetings, and It does 
require meetings. I don't^hlnk that we evaluate enough. 
I think v;e ne^d to have more self evaluation where we do 
soniething like the PQR .. once every six weeks is the way 
I would like to do it. But it seems like we have so many 
things going on at this sch^ool thar require teachers to 

be in minings , So it's very hard to get people together, 

even to g^t a committee together to work on some of th^ese 
things. 'r^Mnk it needs a Tot of Improvement." {13SP2) 

Thts thousht j that much could be ^accompl ished v/1t^ the 
existing evaluation data if only there were time to sit down 
leisurely to study it and make' plans, was voiced by man^ of the 
respondents in our sample. It was pMbably the most clearly 
def 1 ned . encourager to emerge from the interviews. 

In fact^ belief in this proposition was strong enough that 
a few schooTs had attempted to institutionalize greater opportun 
ities for reflection and reorganization'. One school held an 
annual off-campus conference just befpre^the start of the nev/ 
school »year* They selected a comfortable site (neutral turf, 
as it were) where the staff could get together without the 
reaular pressures of school to review the accomplishments of the 
previous year including student test scores and discuss the 
educational activities for' the year to come. Another school 



set aside Its last staff meeting for "reflection and projection" 
during which tlTne the tiacherB could take a more open and 
creative look at the school program and the data avanable 
from the year Just concluded. ' ■ * 

' Unfortunatelyi the two Instances cited abovt^ear to 
be exceptional. Not all schools are taking action to fill the 
need for systematic planning time. But given the existing 
limitations of budgets and calendars It Is not as easy action 
to take. In fact^ the off-canipus conferance eltfd above has 
been reduced from tv/o days to one this year^ and it will be 
held on cam^pus . as wel 1 , The school 's current budget was Just 
too limited to afford the expense of the previous arrangenients . 

If our respondents are to be believed, the Implication for 
utilization I3 clear. More time needs to be provided for reviev/ 
of evaluation data an4 systematic planning based on this infor- 
mation. We make this recommendation with caution^ for the data 
from our study also suggest that the mere existence of pupil 
free afternoons or other open blocks of time will not Insure 
greater attenti on to evaluation. \ There are innumerafile other 

demands competing for such free time. To Increase utilization 

f 

the time should be earmarked in some manner specifically for 
the purpose of analyzing and acting upon evaluation. 

This suggestion is primarily an admin.istrati ve question 
and not something that is within the di rect control of the 
evaluator. However, a know! edgeabl e evaluator might make such 



a suggistion to the decision makers with wh^m he or .s^he is 
working. In facti in some cases an evalua^or can insist on 
a first hand presentation and discussion of the data as the 
chief reporting format. If enough emphasis Is given to this 
demandp then time may be set as1d« for it. ^ 
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Conclusion 

= -f i ' ' ' ' ' \ 

■ . • ■ . . 

Two of the general faat^tres that emerged from our data 
proximity and compitihg demands of time --have baen dis- 
cussed and their action as constraints or encouragers to the 
utilization of evaluation Kas been* analyzed* 

Proximity was divided into structural and temporal^ com- 
portents. The data suggests that Information which is different, 
in form and content from the school's Instructional program 
(structural proximity) is less Tikelx to be used in decision 
making. There are tv/o aspects to temporal proximity. Evalua- 
tion is* 1 ess>^1 kely. to have impact if it comes at a time when 
dec1s1ons^re\^^^ being made (timel iness) and if the 

Information It contains Is no longer current (activ^^ime frame) 
and thus less ^levant to the decision. ^ ^ 

The second general constraining feature that emerged from 
our data v;as competing demands on time. There currently are 
many demands on teacher and administrator time which tend to 
m1n1miz.e the amount of attention that is paid to evaluation. 

This analysis has definite implications for the evaluatot* 
or the school adminlstraf or who is Interested in increasing the 
use of evaluation. When one considers the constraining or 
encouraging potential of the form and content^of evaluatlonj 
the active time fi^n^e of the data, the timeliness of the 
reporting and the competing demands of school personnel ' s time ^ 
a 'number of suggestions for improvement can be made* To increase 
evaluation utilization at the site level the evaluation should be 
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planned so. that: ' • ■ 

!• The dita/ara collected and Teported/ in farm that Is aasy 

^ ' . \ . , . / 

to use and corresponds to whatever organizational systam Is ^ 

■■ » ■ ■ 

in use In the school. . 
2. The Instruments reflect the same content and Internal scope 

as the Instructional program at rthe ichool. 
^3. ,The data collection and reporting process is coordlnate^d 

with, the school calendar and the impor^|gfl|tN^entif iabl^ 

decision periods* 
4/ The data is analyzed and repdrted quickly^ 
5. Time is set aside for reviev/ of the information. In this 

regard^ a flr^ hand presentation v/lth questions, may be much 
fetter |;han a written report. / 



/everj one small caveat 1s in order. The fact that each of 
these factprs was clearly identifiable as a constraint based on 
our data doss not necessarily mean that imprpvement in these 
areas wijl Increase impact. If ^ for example, all evaluation 
were suddeoT^ structur^ally and temporally proxlfnate it might 
be that administrators'^would just point elsewhere to explain 
the continuing non-use of the data, 'One can never know with 
certai nty • t he consequences of suggested changes in the way things 
are done* It is our firm be! i ef , ^ however ^ that the recommendations 
derived from this s tydy wi 11 have positive impact on utilization* 
As a final word, it should be mentionad that our analysis of 



the data 1s not complete. A number of psychosocial variables 



